ASTRID Error Fix
Status:
This documentation applies to version 1.48.849.403 (27/11-2024) of the VdAstrid/CVdAstErrorFix device.
Contents:
- CVdAstErrorFix overview
- Error Fix device
- ASTRID2 error fix scheduler device
- ASTRID automation device
- VdAstrid
- Microtron Modulators Automation
- F0: K100 turning off 'randomly'
- F8: K100 going to standby 'randomly'
- F1: Microtron Sector valve VVS61 and VVS62 Closed
- F2,F16-F18: Microtron VVR61,VVR62,VVR63,VVR64 Roughing valve closed checks
- F13: Inj. Beamline Sector valve VVS03 Closed
- F3-F7: BMH11, QMH11, QMH21,QMV11,QMV21 supply magnet failure
- F9: ASTRID SMH11 failure
- F10: Timing CRIO communication check
- F11: Microtron BMH606 Dipole spike detection/recovery
- F12: InjBl quadrupole error recovery
- F13: VVS03: Inj. beamline Sector valve VVS03 Closed check
- F14: SMH11CIPS turns itself off check
- F15: FMH11IPS turns itself off check
CVdAstErrorFix overview
Device design notes - see document(s) in folder
O:\Nat_ISA-Ast2Doc\machine\Documentation\Controls\Automation\ErrorFix
Microtron Modulators Automation
The Microtron Modulator Error fix methods must only run if the Microtron bunker is closed and has been manual approved to run by operator after bunker closure.
A specific handling of this has been implemented in the device and is controlled by the cluster above.
This device will monitor the safety PLC, PSI60SFTY bunker closed status. If the bunker is opened, microtron modulator automation/error fix procedures should no longer be allowed to run. When blocked,
MicModAuto.automation is FALSE/NotAllowed =>
F0 K100 turning off 'randomly' and
F8: K100 going to standby 'randomly'
can not be trigged
Status:
MicModAuto.condOk - True when conditions for automation is fullfilled (bunker is closed)
MicModAuto.automation - status (Allowed/NotAllowed) maintained by this device
Control:
MicModAuto.setDisable - 'Push' button, Set Mic Modulation allowed to FALSE
MicModAuto.setEnable - 'Push' button, set by operator to enable automation after bunker closed
Source parameters used to monitor bunker closed:
PSI60SFTYmic.ClosedAccess (Must be 'HallClosed')
PSI00SFTYast.uTronD2 (Must be closed)
Remark: MicModAuto.automation going to 'Allowed' trigs automation A0: Microtron Bunker Closed
F0: K100 turning off 'randomly'
EFX00K100ast
Trigger
MachineMode: onAutoRecovery and
Microtron Modulator Automation on and
K100OPmic.stateTargetSet is off and K100OPmic.enableStateTargetOff has not been activated (i.e. a non - intended K100 off).
Action
Set Decaying
Wait up to 30 minutes for K100 active (Communication ok)
Check that MachineMode is still in OnAutoRecovery - if not so, abort action.
Request remote
Reset
Set K100 to Standby
Now wait 30 minutes for stateRead to reach Standby
Check that MachineMode is still in OnAutoRecovery - if not so, abort action.
Record value of MicRFPulse.FwdRF
Reduce CRE61OSC.RFLevel by 2 dBm
Set K100 to Trig
Set A2 beammode to Injection Adjust
Slowly increase CRE61OSC.RFLevel by 0.05 dBm until MicRFPulse.FwdRF is equal to recorded level
Check that MachineMode is still in OnAutoRecovery - if not so, abort action.
Trig a MicAutoAlign(send true to AutoAlignAstInj.AlignNow) and wait until MicAutoAlign is done
If Ok, Restore BeamMode (Filling)
Set MicAutoAlign to run at an interval of 2 h
Action Progress:
0: Inactive
1: Trigged K100 Turn of recovery
2: Run 'Turn of recovery' started:
10000: Detected K100 not active (no communication)
1xxxx: Waiting for K100 active, counting up from 10000. *1
101: K100 active/commicating
102: K100 not active/commicating
103: not ok after K100 active check
104: Machine Mode no longer in OnAutoRecovery (after initial wait)
200: Proceding ok, continue actions
20000: K100 Reset and set to Standby
2xxxx: Waiting for Standby, xxxx seconds. *1
300: Standby ok
301: Standby not reached
302: Machine Mode no longer in OnAutoRecovery (after waiting for standby)
40000: Start adjusting RF forward power
4xxxx: Adjusting RF forward power, steps. *1
400: Machine Mode no longer in OnAutoRecovery (after adjusting RF power)
50000: Starting auto align
5xxxx: Waiting for auto align to finish. *1
600: Setting MicAutoAlign to run at an interval of 2 h
F8: K100 going to standby 'randomly'
EFX00K100STBYast
Trigger
MachineMode: onAutoRecovery and
Microtron Modulator Automation on and
(stateTargetSet going to standby without K100OPmic.enableStateTargetOff enabled) or
(interlocks exists)
Action
Check known to go wrong interlock conditions, Reset K100, Set Trig mode, set Ring operation Topup mode:
Set Decaying
Wait up to 30 minutes for K100 active (Communication ok)
Check that MachineMode is still in OnAutoRecovery - if not so, abort action.
Reset K100
Wait for CCPS.PS1 < 100 V
Check that MachineMode is still in OnAutoRecovery - if not so, abort action.
Set Trig mode
Check recover state - and retry recovery if not ok.
If Recovered, Restore BeamMode (Filling)
Action Progress:
0: Inactive
1: Trigged
2: Checking active status
10000: Detected K100 not active (no communication)
1xxxx: Waiting for K100 active, counting up from 10000. *1
101: K100 active/commicating
102: K100 not active/commicating
103: not ok after K100 active check
104: Machine Mode no longer in OnAutoRecovery (after initial wait)
Recovering, recovery try number 'n'
n000: Waiting for next recover try
n101: Machine Mode no longer in OnAutoRecovery (next revery try)
n100: Resetting K100
n2xx: Waiting for CCPS voltage is below 100 V, xx seconds. *1
n300: Setting K100 trig
n400: Checking for recovered
n500: Machine Mode no longer in OnAutoRecovery (while checking for recovering)
5000: Recovered succesfully
6000: Recovered failed, recovery has been run
7000: Recovered failed, recovery has not been run
F1: Microtron Sector valve VVS61 and VVS62 Closed
EFX00MICVVSast
Trigger
VVS61 Closed or VVS62 closed while MachineMode at least in OnBasicAuto and StatusAst2.BeamMode not in 'Injection Adjust'. The trigger is NOT activated if VVS61ast.close or VVS62ast.close has been pressed before valve close status. In this case, the protection is blocked. A valve open status must be detected to remove blocked condition (manual value open)
Action
Set Decaying
Retry recovery up to 'fixRetries' times
Wait up to 3 minutes for pressures to be OK
Check that MachineMode is still at least in OnBasicAuto - if not so, abort.
'VVS61 to open' = VGD61OK & VGD62OK & VGP63OK & COMP_AIR_OK
'VVS62 to open' = VGD62OK & VGP63OK & VGD64OK & COMP_AIR_OK
'Ok to open' = 'VVS61 to open' & 'VVS62 to open'
If 'Ok to open'
Try opening the valves
Test if valves reopened
End If 'Ok to open'
Check that MachineMode is still at least in OnBasicAuto - if not so, abort further retries.
End Retry recovery
Action Progress:
0: Inactive
1: Trigged
2: Set daying
Recovering, recovery try number 'n'
nxxx: Wait for pressures to be OK, waited xxx seconds (xxx 0-180). *1
n200: Check status
n300: Reopen valves
n400: Test if valves has reopened
n500: Opened failed
n800: Presure not ok to open
n900,n901: Machine Mode no longer at least in OnBasicAuto
3: Not recovered
4: Recovered
F2, F16-F18: Microtron VVR61,VVR62,VVR63,VVR64 Roughing valve closed check
EFX61VVR61ast,EFX62VVR64ast,EFX62VVR64ast,EFX64VVR64ast
Trigger
MachineMode: onAutoRecovery and
VVR6x Roughing valve closed. and
not blocked
The protection is blocked if VVR6xast.close has been pressed before valve close status detected. A valve open status must be detected to remove blocked condition (manual value open)
Action
Set Decaying
Retry recovery up to 'fixRetries' times
Wait up to 30 seconds.
'Ok to open' = VGP63(VVR61,VVR62,VVR63) or VGP64(VGR64) pressure ok and VPT6x running
If 'Ok to open'
Try opening the valve
Test if valve reopened
End If 'Ok to open'
End Retry recovery
Action Progress:
0: Inactive
1: Trigged
2: Set daying
Recovering, recovery try number 'n'
n100: Wait a 5 (first try)/30(following tries) seconds
n400: Check machine mode - failed
n500: Open valve
n600: Conditions for open not fulfilled
n700: Not opened - either responding to open or conditions for open was not fulfilled.
3: Not recovered
4: Recovered
F13: Inj. Beamline Sector valve VVS03 Closed
EFX03INJVVSast
Trigger
VVS03 Closed while MachineMode at least in OnBasicAuto and StatusAst2.BeamMode not in 'Injection Adjust'. The trigger is NOT activated if VVS03.close has been pressed before valve close status detected. In this case, the protection is blocked. A valve open status must be detected to remove blocked condition (manual value open)
Action
Set Decaying
Retry recovery up to 'fixRetries' times
Wait up to 3 minutes for pressures to be OK
Check that MachineMode is still at least in OnBasicAuto - if not so, abort.
'Ok to open' = VGD06OK & VGD64OK & COMP_AIR_OK
If 'Ok to open'
Try opening the valve
Test if valve reopened
End If 'Ok to open'
Check that MachineMode is still at least in OnBasicAuto - if not so, abort further retries.
End Retry recovery
Action Progress:
0: Inactive
1: Trigged
2: Set daying
Recovering, recovery try number 'n'
nxxx: Wait for pressures to be OK, waited xxx seconds (xxx 0-180). *1
n200: Check status
n300: Reopen valves
n400: Test if valves has reopened
n500: Opened failed
n800: Presure not ok to open
n900,n901: Machine Mode no longer at least in OnBasicAuto
3: Not recovered
4: Recovered
F3-F7: BMH11, QMH11, QMH21,QMV11,QMV21 supply magnet failure
EFX00BMH11ast, EFX00QMH11ast, EFX00QMH21ast, EFX00QMV11ast, EFX00QMV21ast
Trigger
If StatusAst2.MachineMode is 'onAutoRecovery' and xxxx.Power is 'On':
Trig if either a supply, phase, supply water, supply temp,earth, main or filter error is detected.
Action
Store Astrid2 operation mode
Set Decaying
Retry recovery up to 'fixRetries' times
Wait for next try
Check that MachineMode is still in OnAutoRecovery - if not so, abort.
Reset Error
Check that MachineMode is still in OnAutoRecovery - if not so, abort.
If 'Ok'
Turn supply off
Wait 2000 ms
Turn supply on
Wait 3000 ms
End If 'Ok'
Check that MachineMode is still in OnAutoRecovery
End Retry recovery
If Ok: Restore Astrid2 operation mode
Action Progress:
0: Inactive
1: Trigged
2: Recovering started, Set Decaying
Recovering, recovery try number 'n'
n001: Waiting for next try
n002: Reset supply and wait
n003: Try to restart supply
n004: Check to see if recovered
n010: Errors still present
n900,n901,902: Machine Mode no longer in OnAutoRecovery
F9: ASTRID SMH11 failure
EFX11SMH11ast
Trigger
Machine mode is 'onAutoRecovery' and SMH11ISP.Power is 'on':
MPS,overload,reg,overcurrent,earth,magnet_water,phase or breaker error is present
Action
Store Astrid2 operation mode
Set Decaying
Retry recovery up to 'fixRetries' times
Check that MachineMode is still in OnAutoRecovery - if not so, abort.
Reset Error
Check that MachineMode is still in OnAutoRecovery - if not so, abort.
If 'Ok'
Turn supply off
Wait 1 second
Turn supply on
End If 'Ok'
Check that MachineMode is still in OnAutoRecovery
End Retry recovery
If Ok: Restore Astrid2 beam mode
Action Progress:
0: Inactive
1: Trigged
2: Recovering started, Set Decaying
Recovering, recovery try number 'n'
n001: Waiting for next try
n002: Reset supply and wait
n003: Try to restart supply
n010: Errors still present
n900,n901,902: Machine Mode no longer in OnAutoRecovery
F10: Timing CRIO communication check
EFX00CRIOast
Trigger
CRIO98 Communication status changed
Action
If communication lost, set Beam Mode decaying
If communication reetablished and decaying mode prev set by communication lost, set prev. beam mode again.
F11: Microtron BMH606 Dipole spike detection/recovery
EFX00BMH606SPKast
Trigger
Machine mode is 'OnAutoRecovery' and
ASTRID automation 'aut2 BMH606IPS Wash' is not running and
BMH606IPSast.Main is 'On' and
Iw has not been changed and
Trig on spike in BMH606IPSast.Ir detection:
A spike is defined as:
(Ir < (Iw - 0.30 A)) OR (Ir > (Iw + 0.10 A)
Action
Set Decaying
Wait 10 seconds
Trigger a BMH606 wash
Wait for 'Aut2: BMH606IPS Wash' finished
If Ok: Restore Astrid2 beam mode
Action Progress:
0: Inactive
1: Trigged
2: Recovering started, Set Decaying
3: BMH606 Wash trigger send
1xxx: Waiting for wash start, xxx seconds
4: Wash in progress detected
2xxx: Waiting for wash finished, xxx seconds
5: Wash finished ok
101: BMH606 wash not finished in time
102: Wash was not started by trig wash
F12: InjBl quadrupole error recovery
EFX00INJBLQUADast
Trigger
Machine mode is 'OnAutoRecovery' and
One of the following is below 0.2 A
QMH63IQP.adc
QMH64IQP.adc
QMV63IQP.adc
QMV64IQP.adc
Action
Test that ConSys connections are OK (no network failure)
If ConSys problems, wait for ConSys recovery
Set Decaying (new from 3/10-23)
Send true to QMx6xREL.reset
Wait 5 s
If one of the parameters is still below 0.2 A then GiveUp
If Recovered: Restore Astrid2 beam mode(new from 3/10-23)
Action Progress:
0: Inactive
1: Trigged
2: Recovering started
1xxx: Waiting for ConSys connections all connected, xxx seconds*1
15xx: Waiting for ConSys feature sources valid, xxx seconds*1
3: QMx6xREL Reset
10x: Wait 5 seconds (x counting down)
2xxx: Check all qmx for recovered (above 0.2 A), xxx seconds*1
5: Recovered ok
500: Action aborted/stopped
501: ConSys connections not ok
502: Inj Bl quad recovery failed (still at least one below 0.2 A)
F14: SMH11CIPS turns itself off check
EFX11SMH11Cast
Trigger
Machine mode is 'OnAutoRecovery' and
Supply status not on (SMH11CIPSast.scPower off)
Action
Scheduler Set BeamMode Decaying
Retry recovery up to 'fixRetries' times
Wait
Check that MachineMode is still in OnAutoRecovery - if not so, abort.
Turn SMH11C Supply On
Wait
Check supply on
End Retry recovery
If Ok: Sheduler Restore Beam Mode
Action Progress:
0: Inactive
1: Trigged
1xxx: Waiting for ConSys connections all connected, xxx seconds*1
15xx: Waiting for ConSys feature sources valid, xxx seconds*1
2: Recovering started, Set Decaying
Recovering, recovery try number 'n'
n0xxx: Waiting for next try xxxx seconds left
n0600: Check machine mode ok.
n0700: Turn SMH11C Supply On
n09xx: Check power status
n0500: Not powered on
2000: Failed ConSys connections not ok
F15: FMH11IPS turns itself off check
EFX11SMH11ast
Trigger
Machine mode is 'OnBasicAuto' and
Supply status not on (SMH11CIPSast.scPower off)
Action
Scheduler Set BeamMode Decaying
Retry recovery up to 'fixRetries' times
Wait
Check that MachineMode is still in OnBasicAuto - if not so, abort.
Reset FMH11 Supply On
Wait
Turn FMH11 Supply On
Wait
Check supply on
End Retry recovery
If Ok: Sheduler Restore Beam Mode
Action Progress:
0: Inactive
1: Trigged
1xxx: Waiting for ConSys connections all connected, xxx seconds*1
15xx: Waiting for ConSys feature sources valid, xxx seconds*1
2: Recovering started, Set Decaying
Recovering, recovery try number 'n'
n0xxx: Waiting for next try xxxx seconds left
n0600: Check machine mode ok.
n0700: Reset SMH11C
n08xx: Wait xx seconds
n0800: If recoving ok (not aborted): Send SMH11C power on.
n09xx: Wait and Check power status
n0500: Not powered on
2000: Failed ConSys connections not ok
*1
ActionProgress: When counting up, only the first and the last value are added to the read/write histories in order to make the histories more readable.
Last Modified 02 December 2024